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EXECUTIVE  SUMMARY 

The  high  cost  of  Weapon  Systems  material  acquisition  in  the  Depart- 
ment of  Defense  (DOD)  has  been  in  the  limelight  on  the  Congressional  and 
National  level  for  many  years.  There  is  an  inherent  requirement  for 
adequate  test  and  evaluation  of  these  weapon  systems  to  ensure  maxi- 
mization of  dollars  spent  for  systems  procured.  DOD  has  shifted  emphasis 
from  Development  Test  and  Evaluation  (DT4E)  to  Operational  Test  and  Evalu- 
ation (0T4E)  by  establishing  Independent  Test  agencies,  who  perform  0T4E 
and  report  test  results  essential  to  key  decisions  for  weapon  system 
acquisition. 

This  report  reviews  existing  directives  and  identifies  concepts  that 
affect  the  balance  of  DT4E  and  0T4E,  which  concern  Naval  Air  Systems 
Command  (NAVAIR)  Project  Managers.  Interviews  with  NAVAIR  Project  Managers 
eind  Chief  of  Naval  Operations  (OP-983)  were  the  source  for  current  data. 

This  study  revealed  that  the  Navy  org2uiization  has  been  stable, 
efficient,  and  effective  for  the  Procurement  of  Weapon  Systems,  throughout 
the  period  prior  to  and  since  the  inception  of  the  new  DOD  policy.  There 
is  little  or  no  overtesting  being  performed  at  this  time  and  the  emphasis 
that  the  Navy  places  on  the  requirement  for  a Test  and  Evaluation  Master 
Plan,  at  the  earliest  decision  milestone,  is  an  effective  method  to  pro- 
duce a balance  of  DT4E  and  0T4E. 

The  most  significant  changes  required  for  improvement  are  establish- 
ment of  an  effective  T4E  cost  accounting  tracking  system  at  Navy  agencies, 
and  high  level  interest  to  support  sufficient  funding  for  the  design  phase 
of  each  approved  weapon  system  acquisition. 
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SECTION  I 


INTRODUCTION 

Piirpose  of  the  Study  Pro.iect 

The  high  cost  of  Weapon  Systems  material  acquisition  in  the  Depart- 
ment of  Defense  (DOD)  has  been  in  the  lime-light  on  the  Congressional  and 
National  level  for  many  years.  In  times  of  War  the  emphasis  wanes  to  a 
certain  degree,  but  when  conflict  ends,  attention  is  once  again  heavily 
focused  not  only  on  the  military  need,  but  on  the  high  price  of  procure- 
ment and  operation.  The  Inherent  requirement  for  adequate  test  and  evalu- 
ation of  these  weapon  systems  to  ensure  maximization  of  dollars  spent  for 
systems  procured,  has  been  a major  concern  and  emphasized  since  1970. 
Additionally , the  rising  cost  of  the  attendant  test  and  evaluation  process 
in  an  economy  such  as  we  live  in  today  further  perturbates  this  concern. 

In  an  effort  to  provide  a balanced  cost  effective  test  and  evalu- 
ation program  for  all  weapon  systems  acquisition,  DOD  has  shifted  their 
emphasis  from  Development  to  Operational  test  zuid  evaluation,  by  estab- 
lishing Independent  test  agencies  who  provide  operational  testing  and 
reports  essential  to  the  decision  making  process. 

This  report  reviews  the  existing  DOD  and  Navy  directives  and  policy 
that  have  been  published  and  are  of  import  within  the  Naval  Air  Systems 
Command  (NAVAIR).  It  will  identify  emerging  concepts  that  affect  the  test 
amd  evaluation  process  to  describe  a balance  or  imbalance  in  our  present 
test  methodology.  Additionally,  it  will  describe  the  effect  that  this 
emphasis  on  Operational  test  and  evaluation  has  had  on  the  NAVAIR  Project 
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Manager  and  the  acquisition  process. 
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Additional  change  and  shift  of  emphasis  in  the  test  and  evaluation 
Community  will  certainly  occur,  therefore,  it  is  hoped  that  this  report 
will  help  to  generate  future  studies  to  identify  more  efficient  and  cost 
effective  test  and  evaluation. 


Goals  of  the  Project 

Specific  goals  for  this  report  aret 

1.  To  research  and  review  the  current  directives  and  policy  for 
test  and  evaluation  as  delineated  by  DOD  and  as  they  apply  to  the  Navy. 

2.  To  research  pertinent  reports,  factors,  and  criticisms  »dilch 
have  determined  the  current  course  of  action,  and  structure  of  T&E 
emphasis  in  the  Navy. 

3.  To  determine  the  existance  of  overtesting  or  Inadequate  testing 
in  respective  areas. 

4.  To  investigate  selected  NAVAIR  projects  to  determine  the  impact 
of  Operational  Test  and  Evaluation  (OTE)  on  the  Development  process. 

5.  To  determine  the  Impact  that  OTE  has  on  the  acquisition  process 
and  the  level  of  difficulty  in  obtaining  approval  for  Service  use. 

6.  To  examine  the  extent  and  effectiveness  of  integrated  testing. 

7.  To  make  recommendations  to  Influence  policy  and  future  direction 
of  Test  and  Evaluation. 
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Scope  zmd  Limitations 


This  study  effort  was  limited  by  the  time  available  to  conduct  the 
Individual  Study  Program  at  the  Defense  Systems  Maneigement  College.  It 
consisted  of  research  of  the  directives,  reports,  and  policy  statements 
that  affect  the  current  posture  of  the  DOD  policy  on  Test  amd  Evaluation. 

A survey  of  Navy  instructions,  related  policy  statements,  and  briefing 
materials  was  also  performed.  Interviews  with  present  and  former  NAVAIR 
Project  Managers  (PM)  and  Acquisition  Managers  (AM)  were  conducted  to 
btain  current  views,  impressions  and  impact  responses  to  the  OTE  emphasis 
their  projects.  A non-attribution  guarantee  was  provided  each  inter- 
viewee to  obtain  the  most  candid  impressions  possible.  Travel  to  OPTEVFOR, 
Norfolk,  Va.,  was  contemplated,  but  rejected  as  time  becajne  premium.  In 
lieu  of  this  effort  an  interview  with  Chief  of  Naval  Operations  (CNO) 

OPNAV  Code  983  personnel  who  represent  OPTEVFOR  was  accomplished.  The 
term  of  reference  for  this  report  therefore  is  biased  to  the  DTE  side  of 
the  acquisition  process. 


Definitions 


In  order  to  discuss  what  composes  the  term  balance  of  test  and  evalu- 
ation, it  is  appropriate  here  to  present  two  basic  definitions  of  Develop- 
ment Test  and  Evaluation  (DT4E)  and  Operational  Test  and  Evaluation  (0T4E) 
respectively.  DOD  Instruction  5000. 3 provides  the  baseline  for  the  myrieui 
of  Navy  directives  that  have  been  published  to  Implement  policy. 

Development  Test  and  Evaluation  (PTcfeE).  DT4B  Is  that  test 
and  evaluation  conducted  toi  demonstrate  that  the  engineering 
design  and  development  process  is  complete t demonstrate  that 


3 


the  design  risks  have  been  minimized;  demonstrate  that 
the  system  will  meet  specifications;  and  estimate  the 
system's  military  utility  vrtien  introduced.  DT&E  is 
planned,  conducted,  and  monitored,  by  the  developing 
agency  of  the  DOD  component,  and  the  results  thereof 
are  reported  by  that  agency  to  the  responsible  Mili- 
tau7  Service  Chief  or  Defense  Agency  Director.  (1«2)^ 

Operational  Test  and  Evaluation  is  more  heavily  emphasized  in  the 

basic  document,  DOD  Instruction  5000.3  and  specifically  states  that, 

"operational  testing  should  be  separate  from  development  testing".  (l«4) 

The  definition  of  OT&E  then  follows i 

Operational  Test  and  Evaluation  (OT&E).  OT&E  is  that 
test  and  evaluation  conducted  to  estimate  the  prospec- 
tive system's  military  utility,  operational  effective- 
ness, and  operational  suitability  (including  compati- 
bility, interoperability,  reliability,  maintainability, 
eind  logistic  and  training  requirements),  and  need  for 
any  modification.  In  ciddition,  OT&E  provides  infor- 
mation on  organizations,  personnel  requirements,  doc- 
trine, eind  tactics.  Also  it  may  provide  data  to 
support  or  verify  material  in  operating  instructions, 
publications,  and  handbooks.  OT&E  will  be  accomplished 
by  operational  amd  support  personnel  of  the  type  and 
qualifications  of"  those  expected  to  use  and  maintain 
the  system  when  deployed,  and  will  be  conducted  in  a 
realistic  and  operational  environment  as  possible. 

OT&E  will  normally  be  conducted  in  phases,  each  keyed 
to  an  appropriate  decision  point.  During  Full  Scale 
Development  OT&E  will  be  accomplished  to  assist  in 
evaluating  operational  effectiveness  and  suitability 
(including  compatibility,  interoperability,  relia-  , 
bility,  maintainability,  and  logistic  and  training 
requirements).  OT&E  will  be  continued  as  necessary 
during  and  after  the  production  period  to  refine 
these  estimates,  to  evaluate  changes,  and  to  re- 
evaluate the  system  to  Insure  that  it  continues  to 
meet  operational  needs  auid  retains  its  effective- 
ness in  a new  environment  or  against  a new  threat. 

(1«3) 


1 This  notation  will  be  used  throughout  the  report  for  source  of 
quotations  and  major  references.  Ibe  first  number  is  the  sources 
listed  in  the  bibliography.  The  second  number  is  the  page  in  the 
reference.  Author  underlining  for  emphasis. 
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SECTION  II 


BACKGROUND  AND  CURREm  POLICY 

On  1 July  1970,  the  Report  to  the  President  and  the  Secretary  of 
Defense  on  the  Department  of  Defense,  by  the  Blue  Ribbon  Defense  Panel, 
was  published.  It  was  this  catalyst  which  caused  the  significant  chainge 
in  the  test  ajnd  evaluation  process  for  acquisition  of  weapon  systems. 

Prior  to  1970  congressional  pressure  (more  so  in  the  50 's  than  in 
the  60's)  had  not  demanded  such  strict  accounting  and  attention  to  details 
of  each  test  and  evaluation  process,  that  they  would  exert  later  in  the 
70’s.  They  seemed  liatisf led  until  the  late  60's  when  their  attention 

k 

was  shifted  toward  the  increasing  cost  of  procurement  and  deployed 
operation  of  Defense  Weapon  Systems.  The  Blue  Ribbon  Defense  Panel  was 
appointed  by  the  President  in  1969  to  conduct  an  indepth  review  of 
operation  of  the  Department  of  Defense,  which  included  a detailed  analyses 
of  test  and  evaluation  in  the  component  Service  organizations.  The 
"Fitzhugh  Report"  does  not  specifically  mention  or  use  the  term  DT&E 
in  the  discussion  on  Operational  Test  and  Evaluation,  but  it  is  important 
to  recognize  that  a differentiation  between  "functional"  and  "operational" 
testing  is  madet 

Functional  testing  (often  called  engineering  testing)  is 
done  to  determine  how  well  various  systems  and  material 
meet  design  and  performance  contractual  specifications 
in  other  words,  whether  they  meet  technical  requirements. 

By  and  large,  functional  testing  in  and  for  the  Depart- 
ment of  Defense  appears  to  be  well  xinderstood  and  faith- 
fully executed.  Serious  policy  deficiencies  are  not 
apparent,  and  such  failures  in  functional  testing  as 
occur  can  be  primarily  attributed  to  lack  of  technical 
competence,  oversight,  or  procedural  breakdowns. 


5 


Operations . auid  (2)  there  is  a formal  way  of  getting 
operational  evaluation  (including  some  operational 
testing)  done  early  in  the  overall  process.  The 
main  deficiency  in  Navy  OT&E  is  that  it  generally 
produces  few  haurd  data. 

The  staff  report  on  OT&E  concluded  that* 

OT&E  within  the  Services  is  done  most  effectively 
vdien  OT&E  organizations  report  directly  to  the  Chief 
of  the  Service,  representing  both  the  developer  and 
user,  but  organizationally  Independent  of  both. 

There  are,  however,  considerable  forces  within  the 
Service  which  resist  the  Independence  of  OT&E  organi- 
zations. (3*7) 

The  Navy  method  seemed  to  be  the  recommended  approach  vdilch  left  only 
emphasis  on  independence  and  early  involvement  as  the  key  Issues  to 
persue  in  the  future  towards  creating  a balance  of  DT&E  and  OT&E. 

Deputy  Secretary  of  Defense  Paskazd  was  the  driving  force  behind 
the  DOD  shift  in  emphasis  from  the  concept  of  Total  Package  Procurement 
to  Milestone  Procurement  for  Defense  Weapon  Systems.  He  issued  a memo- 
randum to  the  Services,  on  February  11,  1971  which  responded  to  the 
"Fitzhugh  Report”  and  stated j 

Althou^  each  Service  now  has  a somewhat  different 
way  of  organizing  for  operational  test  and  evalu- 
ation, it  is  apparent  to  me  that  this  function  can 
best  be  performed  by  an  agency  which  is  separate 
and  distinct  from  the  developing  command  and  which 
reports  the  results  of  its  test  and  evaluation 
‘efforts  directly  to  the  Chief  of  the  Service.  (4) 

This  was  precisely  what  the  Navy  was  doing  as  perceived  by  the  "Fltzhu^ 

Report"  staffers.  He  went  on  to  state  that: 

...within  the  service  headquarters  staff,  there  needs 
to  be  an  office  with  a clear  OT&E  identification  to 
provide  staff  assistance  directly  to  the  Service  Chief 
and  to  provide  a headquarters  focal  point  for  the  inde- 
pendent OT&E  field  agency. . .each  service  is  requested 
to  restructure  its  organization  for  OT&E  along  the 
lines  specified  above.  (^) 
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Commander 


Since  the  Navy  already  had  am  Independent  OT&E  test  agency  In 
Operational  Test  and  Evaluation  Force,  COMOPTEVFOR  at  Norfolk,  Virginia, 
the  remaining  directive  action  would  be  accomplished  by  subsequent 
establishment  oi  the  Chief  of  Naval  Operations  in-house  office 
designated  OP-983.  COMOPTEVFOR  was  appointed  director  of  this  office. 

The  Deputy  Secretary  of  Defense  also  established  a Deputy  Director  for 
Test  and  Evaluation  within  DDR&E. 

The  concept  of  Milestone  Procurement  was  formally  established  in 
1971  by  the  issuance  of  DOD  Instruction  5000.1  and  has  been  updated 
twice  since  that  event.  OT&E  and  specifically  the  Operational  Evaluation 
are  mandatory  prior  to  the  Production  Milestone  III  and  emj^aslzed  as 
follows  I 

Test  eind  evaluation  shall  commence  as  early  as  possible. 

An  estimate  of  military  utility  and  of  operational 
effectiveness  amd  operational  suitability  Including 
logistic  support  requirements,  shall  be  made  prior  to 
large  scale  production  commitments.  The  most  realistic 
test  environment  possible  and  acceptable  representation 
of  the  future  operational  system  will  be  used  in  the 
testing, . . . (5*8) 

The  Congress  had  also  had  an  opportvinlty  to  digest  the  thrust  of  the 

"Fitzhugh  Report"  which  fit  in  line  with  their  desires  and  in  1971 

added  sectio.  506  to  the  1972  Authorization  Act  which  stated t 

Beginning  with  1972,  the  Secretary  of  Defense  shall 
submit  to  the  Congress  each  calendar  year,  at  the 
same  time  the  president  submits  the  Budget  to  Congress, 
a written  report  regarding  development  and  procurement 
scheduled  for  each  weapon  system  for  which  fund  authori- 
zation is  required  and  for  which  any  funds  for  procurement  • 
are  requested  in  such  budget.  Beginning  with  the  calendar 
year  1973,  there  shall  be  inclixied  in  the  report  data  on 
operational  testing  and  evaluation  for  each  such  weapon 
system  for  which  funds  for  procurement  are  requested... 
the  results  of  all  operational  testing,  or,  if  operational 
testing  and  evaluation  has  not  been  conducted,  a statement 
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of  the  reasons  therefore,  amd  the  results  of  such 
other  testing  amd  evaluation  as  has  been  conducted,  (6) 

It  is  interesting  to  note  that  SECNAVINST  5OOO.I  of  I3  March,  1972 
was  published  before  DODINST  5000. 3 of  19  January  1973 » which  cleaned  up 
a myriad  of  directives  and  established  policy  in  tune  with  DOD  directives. 
CTiE  is  stressed  in  SECNAVINST  5000.1  but  seems  to  be  in  proper  perspec- 
tive to  the  predetermining  DT&E.  "Test  and  Evaluation  effort  shall  be 
effectively  correlated  with  previously  outlined  requirements  concerning 
approval  of  material  for  service  use."  This  directive  presented  a 
I*iilosophy  for  "Balanced"  test  and  evaluation  in  stating t 

The  wide  variety  of  naval  weapons  dictates  varying 
approaches  to  the  conduct  of  test  and  evaluation; 
such  effort  shall  be  tailored  to  the  needs  and 
characterizations  of  each  individual  acquisition — 
prime  consideration  being  given  to  adequate  opera- 
tionally oriented  testing.  (7tl3) 

New  acquisitions  Includes  conversions,  major  modi- 
fication ajid  modernizations.  Adequate  test  and 
evaluation  of  these  is  also  required  to  (l)  support 
design  and  development  activity,  (2)  measure  per- 
formance against  specified  acceptance  criteria,  (3) 
ensure  satisfactory  operation  with  related  inter- 
facing systems,  (4)  confirm  operational  effective- 
ness and  suitability,  and  (5)  validate  the  adeqtiacy 
of  documentation  for  support  and  test  equipment, 
personnel,  training,  maintenance  amd  operation  of 
the  whole  systera/subsystems , and  other  elements.  (7«14) 

DOD  Test  and  Evaluation  policy  is  given  in  DOD  Directive  5000*3 

defines  that  "The  DD(T&E)  has  across-the-board  responsibility  for  OSD 

in  test  and  evaluation  matters."  More  specifically  DT&E  amd  OT&E 

policy  is  thus  stated 1 

1.  DT&E  shall  be  started  as  early  in  the  develop- 
ment cycle  as  possible  amd  Include  testing  of 
component(s),  subsystem(s) , amd  prototype  or  pre- 
production  model(s)  of  the  entire  system.  Compati- 
bility and  interoperability  with  existing  or  plamned 
equipments  amd  systems  shall  be  tested. 
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2.  During  the  development  phase. . .adequate  DT&E  shall 
be  accomplished  to  demonstrate  that  technical  risks 
[ have  been  Identified... 

3*  During  full  scale  development  jrfiase. . .DTE. . . to 
Insure ...  engineering  complete...  significant  design 
problems  (including  compatibility,  Interoperability, 
i reliability,  maintainability,  and  logistical  con- 

siderations) have  been  Identified | and  that  solutions . . . 
are  In  hand.  (lt2) 


Concerning  Operational  Test  4 Evaluation  policy: 

1.  In  each  DOD  component  there  will  be  one  major 
field  agency  seperate  and  distinct  from  the  developing/ 
procuring  command  and  from  the  using  command  which  will 
be  responsible  for  0T4E  and  which  will: 

a.  Report  the  results  of  independent  test  ajid 
evaliation  directly  to  the  Military  Service  Chief 
or  Defense  Agency  Director. 

b.  Recommend. . .the  accomplishment  of  adequate  0T4E. 

c.  Insure  that  0T4E  is  effectively  planned  and 
conducted.  (1:3) 


Concerning  Production  Acceptance  Test  and  Evaluation  (PAT4E),  which  for 

the  Navy,  Is  conducted  by  the  Board  of  Inspection  and  Survey  (BIS),  for 

major  weapon  systems  and  Is  defined  with  the  following  policy: 

PAT4E  is  test  and  evaluation  of  production  items  to 
demonstrate  that  the  Items  procured  fulfill  the  re- 
quirements and  specifications  of  the  procuring  con- 
tract or  agreements.  It  Is  the  responsibility  of 
each  DOD  component  to  accomplish  the  necessary  PAT4E 
throxighout  the  production  phase  of  the  acquisition 
process.  (1:6) 


j The  history  of  the  Board  of  Inspection  and  Svirvey  (BIS)  dates  back 

I almost  to  beginning  of  the  Navy  in  these  United  States  and  has  functioned 

I by  charter  in  the  same  manner,  but  for  this  purpose  PIS: 

I 

I 


i 
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...is  a shore  activity .. .under  the  direction  of  a 
President. . .under  the  Chief  of  Naval  Operations. 
The  Sub-Board  of  Inspection  and  Survey  office  at 
Patuxent  River  functions. . .under  the  Board  of 
Inspection  and  Survey i Washington,  D.C. 

The  mission... is  to  conduct  acceptance  trials 
of .. .aircraft. . .to  make  recommendations  on  their 
acceptance  by  the  Navy  (8:2) 


The  Operational  Test  and  Evaluation  Force  (OPTEVFOR)  was  established 

during  World  War  II  and  concentrated  in  the  area  of  "tactics"  for  many 

years,  supporting  the  NAVAIR  through  "development  assists"  (d/v)  or 

flight  testing.  In  fact  it  was  common  on  Anti-Submarine  Warfare  (ASW) 

projects  to  obtain  a (d/v)  from  CNO  to  do  a TEChKVAL  followed  by  OPEVAL 

on  the  same  test  aircraft.  OPTEVFOR  is  now  chartered  with  the  stamp  of 

independent  testing  authority  as  follows: 

To  operationally  test  and  evaluate  specific  weapon 
systems. . .aircraft  and  equipments  including  pro- 
cedures and  tactics,  where  required;  and,  when 
directed  by  CNO,  assist  development  agencies  in 
the  accomplishment  of  necessary  Development  Test 
and  Evalmtion. 

...carry  out  assigned  responsibilities  as  an  inde- 
pendent test  agency  for  required  operational  test 
and  evaluation  under  the  command  of  CNO  and  serve 
as  principal  advisor  to  CNO  for  all  Department 
of  the  Navy  laatcer  pertaining  to  operational  test 
and  evaluation.  (9sl) 

Provide  the  results  of  operational  test  and  evalu- 
ation to  Defense  Systems  Acquisition  Review  Council 
(DSARC)  production  decision  reviews. . .and  to  other 
reviews  as  required.  (9:2) 


Approval  for  Service  Use  (ASU)  is  the  bottom  line  at  the  end  of  the 
T4E  process  when  a production  go-ahead  is  obtained.  DT&E  and  OT&E  success 
supports  Milestone  III  Production  decision.  ASU  in  the  Navy  is: 
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That  determination  made  by  the  Chief  of  Naval 
Operations,  or  other  delegated  authority,  that 
new  systems  or  equipment  or  significant  alteration 
to  existing  systems  or  equipments  have  undergone 
appropriate  test  and  evaluation . . . (10«4) 


Provisional  approval  for  Service  Use  (PASU)  can  be  granted  after  under- 
going early  phases  of  appropriate  T&E  in  conformance  with  the  directive. 
Additionally  j 

A "provisional  approval"  may  be  sought  on  programs 
for  which  sufficient  operational  testing  to  support 
a final  determination  for  approval  for  service  use 
cannot  practically  be  accomplished  prior  to  making 
an  initial  production  commitment.  (10«5) 


The  direction  for  acceptance  for  service  use  rests  with  the  BIS  report 
andt 

The  determination  made  by  SECNAV  to  accept  or  re- 
ject an  aircraft  model ... based  on  the  results  of 
an  acceptance  trial  conducted  by  the  Board  of 
Inspection  and  Survey  (BIS).  The  report  of 
BIS  includes  a statement  or  finding  that  an . . . 
aircraft  model  is  recommended  for  acceptance 
or  rejection,  based  on  a complete  inspection 
and  operational  trial  of  the  production  article.  (I0i6) 


When  OPNAV  Instruction  3960.8  dated  22  January  1973  published  many 
Navy  personnel  felt  that  the  T&E  policy  was  then  firmly  established,  and 
that  a "balanced"  test  and  evaluation  process  had  been  instituted.  How- 
ever, a key  issue  began  to  emerge  at  high  levels  in  CNO  >diich  would  cause 
a significant  change  in  1975 • It  centered  around  the  Test  and  Evaluation 
Master  Plan  (TEMP),  which  the  Navy  considered  a key  planning  and  control- 
ling T&E  document.  The  issue  was,  "Who  should  approve  the  TEMP?"  OPTBVFOR 
also  OP-983 » under  the  Chief  of  Naval  Operations  was  driving  this  issue, 
because  the  TEMP  was  being  approved  by  the  Project  Manager,  who  had  re- 
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sponsibility  for  funding  T4E  up  to  the  Major  Production  Milestone  III. 
On  22  October  1975 » OPNAV  Instruction  3960.10  was  published,  as 
authored  by  OP-983,  with  TEMP  approval  vested  in  the  program  sponsor 
(DCNO),  who  will  resolve  conflicts  between  the  Developing  Agency  and 
COMOPTEVFOR . "Navy  policy  is  the  same  as ... " that  stated  in  DOD 
directive  5000. 3.  In  addition  the  main  purpose  of  OPNAVINST  3960.10 
is  to: 


•■Establish  policy  for  test  zind  evaluation  (T&E)  in  the 
Navy  acquisition  programs. 

. Define  the  T&E  responsibilities  of  CNO,  Director  RDT&E 
(OP-98),  prograim  sponsors  (DCNOS/DMSOS),  CHNAVMAT, 
Developing  Agencies  (DAs),  COMOPraVFOR,  PREINSURV,  and 
fleet  commanders. 

. Establish  procedures  for  planning,  conducting,  and 
reporting  T&E. 

. Delineate  the  complimentary  relationship  of  develop- 
ment T&E  and  operational  T&E  phases  throughout  the 
life  of  a program. 

. Establish  procedures  for  obtaining  Fleet  RDT&E 
Support  for  R&D  that  is  not  part  of  the  acquisition 
program,  (llil) 


The  establishment  of  Acquisition  Categories  was  a significant  part  and 
is  the  basis  for  the  various  phases  of  T&E  in  the  Navy.  Figure  1, 
describes  these  acquisition  categories,  however,  as  a result  of  DOD 
directive  5000.1  dated  18  January  1977,  major  program  dollar  thresholds 
have  been  raised  to  $75  million  RDT&E  juid  $300  million  production.  Figure 
2,  displays  the  various  DT&E  and  OT&E  relationships  and  phasing  concept. 
The  Technical  Evaluation  (TECHEVAL)  and  Operational  Evalviation  (OPEVAL) 
will  be  scheduled  back-to-back  to  inhibit  concurrent  testing  prior  to 
the  Production  Decision  Milestone  III.  Additional  information  describing 
these  phase  concepts  should  be  persued  by  the  reader. 
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SECTION  III 


TEST  AND  EVALUATION  ORGANIZATION 

The  T&E  organizational  relationships  seem  to  be  clear  cut  down  to 
the  Naval  Air  Systems  Command  (NAVAIR)  level,  where  some  departure  from 
a formal  chain  reporting  system  takes  place.  Deputy  Director  Test  and 
Evaluation  (DDT&E)  is  the  top  T&E  official  who  reports  to  the  Secretary 
of  Defense.  From  DDT&E  who  has  an  informal  chain  to  OP-983,  the 
Secretary  of  the  Navy  (SECNAV)  controls  RDT&E  and  policy  for  the  Navy, 
down  through  the  Chief  of  Naval  Operations  (CNO),  and  also  has  a vested 
interest  in  the  Navy  Laboratories,  who  conduct  Basic  Research  and  support 
testing  (DT&E)  up  to  the  full  scale  development  phase  of  weapon  systems 
projects.  CNO  down  through  the  Chief  of  Naval  Material  (NAVMAT),  pro- 
vides guidance  and  policy  for  the  execution  of  Navy  projects.  CNO  must 
asses  the  DT&E  and  OT&E  results  to  determine  Approval  for  Service  Use 
(ASU)  to  sustain  the  production  requirement  and  the  Five  Year  Defense 
Plan  (FYDP).  OP-98  Is  the  focal  point  for  T&E,  and  OP-983  under  him, 
who  is  also  Commander,  Operational  Test  and  Evaluation  Force  (COMOPTEVFOR ) 
for  OT&E.  In  turn,  the  actual  performance  of  OT&E  and  OPEVAL  is  conducted 
by  the  squadrons  assigned  COMOPTEVFOR.  It  is  this  chain  that  can  pro- 
vide OT&E  and  OPEVAL  test  results  as  high  as  DDT&E  through  the  informal 
link  without  Prograjn  Manager  awareness,  as  has  been  the  case  on  some 
occasions  on  certain  projects. 

The  second  leg  of  this  triumvirate  T&E  "Czarship"  is  through 
NAVMAT,  who  also  has  an  interest  in  the  Navy  laboratories,  down  to  the 
Commander,  Naval  Air  Systems  Command  (NAVAIR ),  who  has  field  activities 
to  perform  DT&E  as  required  by  the  Projects  assigned  to  the  Command, 
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TECHEVAL  can  be  performed  by  the  laboratories,  but  usually  in  the  form 
of  Naval  Preliminary  Evaluations  (NPE)  in  the  case  of  aircraft,  or  Naval 
Technical  Evaluations  (NTE)  in  the  case  of  missiles,  it  is  done  by  the 
NAVAIR  field  activities. 

The  last  T&E  "Czar"  is  the  President,  Board  of  Inspection  and 
Survey  (BIS),  who  reports  and  is  chartered  by  CNO,  in  addition  he  reports 
BIS  test  results  to  SECNAV.  In  the  acquisition  process  of  Weapon  Systems 
for  NAVAIR  projects,  BIS  plays  an  important  role  to  determine  Service 
acceptability.  As  a sub  board,  (BIS)  at  the  Naval  Air  Test  Center  (NATC) 
can  call  on  NAVAIR  field  activities  to  do  Production  Acceptance  Test  and 
Evaluation  (PAT&E)  for  which  he  is  chartered.  BIS  trials  are  usually 
performed  as  a final  phase  of  T&E,  following  the  last  NPE  or  NTEj  BIS 
acts  as  an  independent  agent  responsible  to  report  deficiencies  to  SECNAV, 
CNO,  and  NAVAIR. 

Figure  3»  depicts  the  organization  of  Navy  T&E,  specifically  through 
the  Chain  of  Command  of  NAVAIR . 

Today  in  NAVAIR  there  is  an  Assistant  Commander  for  Test  and  Evalu- 
ation (AIR-06),  a flag  officer,  but  is  not  depicted  on  Figure  3 to  avoid 
clutter.  The  establishment  of  AIR-06  in  May  1975 » is  the  culmination  of 
events,  that  started  in  196?  with  an  ADHOC  study  which  caused  the  COM- 
NAVAIR  to  create  a management  group  called  the  NAVAIR  T&E  Coordinator. 

The  main  charter  fiaictlons  were  resource  management,  (principally  air- 
craft), long  range  planning  for  NAVAIR  field  activities,  and  T&E  planning 
and  resource  allocation  for  NAVAIR  projects.  Today  these  functions  con- 
tinue, but  AIR -06  has  new,  higher  order,  responsibility  for  management 
and  control  of  the  Budgets  for  NAVAIR  field  activities,  and  the  Navy 
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ranges,  which  are  part  of  the  Major  Range  and  Test  Facility  Base.  As  the 
focal  point  in  NAVAIR  for  T&E  policy  and  resources,  close  liaison  is 
maintained  with  DDT&E  to  vdiom  he  reports  for  NAVAIR  on  Budget  matters. 

An  additional  fimction  concerns  the  development  and  processing  in  the 
approval  chain  of  the  TEMP.  AIR-06  is  unique  in  his  role  within  the 
T&E  process  and  will  surely  be  involved  in  new  issues  that  can  improve 
the  balance  of  DT&E  and  OT&E. 

DOD  satisfaction  that  the  services,  and  specifically  the  Navy, 
have  sound  organizational  structure  to  conduct  "balanced"  test  and  evalu- 
ation is  evidenced  when  DDT&E  reported  to  Congress  in  March  1975 » thati 

Each  of  the  three  Services  now  has  a strong,  mature 
organization  for  the  accomplishraer't  of  Independent 
operational  test  and  evaluation.  (12:3) 

Dr.  Currie,  DDR&E,  in  February  1976  reported  to  the  Congress  that: 

High  level  emphasis  on  T&E  was  assured  initially 
by  creation  of  the  Office  of  Deputy  Director  of 
Defense  Research  and  Engineering  (Test  and 
Evaluation). . .this  office  has  closely  monitored 
the  T&E  of  all  major,  and  many  less  than  major, 
weapons  acquisition  programs.  Further  strength- 
ening of  DOD  T&E  has  been  achieved  by  establish- 
ment of  independent  operational  T&E  agencies 
within  each  of  the  services.  Creation  of  these 
agencies  has  had  significant  impact  on  systems 
development  and  procurement  as  evidenced  by  the 
greater  attention  being  directed  to  operational 
effectiveness  within  the  Services  and  by  the 
greatly  improved  quality  of  Tj^  dato  now  being 
presen  ted  to  the . . . (dSABC ) . (13»2) 

No  amticipated  changes  in  the  Navy,  auid  specifically  the  NAVAIR  organi- 
zation are  anticipated  beyond  those  presented  above. 
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SECTION  IV 


TEST  AND  EVALUATION  MASTER  PLAN 

DOD  emphasis  on  the  need  for  a TEMP,  which  is  required  for  each 

key  milestone,  directs  that: 

The  DOD  component  will  prepare  as  early  as  possible 
in  the  acquisition  process,  eind  prior  to  initiation 
of  Full-Scale  Development,  an  overall  test  and 
evaluation  plan  to  identify  and  integrate  the  effort 
and  schedules  of  all  T&E  to  be  accomplished  and  to 
insure  that  all  necessary  T&E  is  accomplished  prior 
to  key  decision  points.  The  TEMP  will  be  kept 
current  by  the  DOD  component.  (1:6) 

COMOPTEVFOR  is  in  a powerful  position  as  a result  of  the  emphasis  on 

OT&E  since  the  publication  of  the  "Fitzhugh  report",  and  pressure  that 

Congress,  DOD,  and  the  Navy  exerted  on  compliance  with  OT&E  requirements. 

He  addressed  the  CNO  Executive  Board  on  March  25 t 1975?  "On  the  weakness 

in  current  Navy  procedures  that  were  detracting  from  the  Navy's  proper 

and  complete  implementation  of  OT&E  and  the  directives  which  regulate 

it.  He  cited  the  lack  of  adequate  planning  by  Project  Managers  for 

OT&E."  (14) 

It  was  through  OP-983  (COMOPTEVFOR)  that  the  new  Navy  instruction 
on  T&E,  OPNAV  instruction  3960.10,  was  published.  No  significant  changes 
to  the  "Navy  way"  for  acquisition  T&E  were  apparent,  with  the  exception 
of  the  belabored  detail  of  how  to  prepare  a TEMP.  The  impact  was  a great 
deal  more  time  and  effort  that  is  required  to  put  one  together,  amd  the 
approval  chain  process.  The  connotation  that  "balanced"  T&E  exists,  is 
reflected  by  the  "position  power"  of  OP-983/COMOPTEVFOR),  in  that  the 
position  of  the  TEMP  submitted  by  COMOPTEVFOR  is  "sacred",  and  cannot  be 
tampered  with  unless  approved  by  the  CNO,  OP-98,  who  oversees  both  DT&E 
and  OT&E.  Ihe  author  is  personally  aware  that  many  heated  debates  have 
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occured  over  the  Issues,  but  in  most  cases  were  caused  by  personality 
situations,  amd  were  negotiated  in  the  end  analysis.  In  effect  the 
TEMP  forces  the  Project  Manager  to  do  his  "homework"  and  ensure. that 
a mutually  agreed  TEMP  is  derlY«d  at  the  working  level  early  in  the 
conceptual  phase. 

"The  TEMP  is  the  controlling  management  document  which  defines  the 
test  and  evaluation  for  each  acquisition  program... it  contains  the  inte- 
grated requirements  of  DT&E  and  OT&E,  and  the  schedule  amd  resources 
required  for  accomplishment.  The  TEMP  will  be  prepared. . .and  approved 
prior  to  Milestone  I.  The  TEMP  will  be  prepared  by  the  DA  in  cooperation 
with  COMOPTE'/FOK  and  PRESINSURV  (BIS).  Approval  of  the  TEMP  constitutes 
CNO  direction  to  conduct  the  T&E  program..."  (11:?) 

The  Program  F'anjiger  is  the  one  who  must  perform  this  "balance" 
act  for  the  T&E  he  knows  is  necessary  for  his  project,  but  is  constrained 
by  the  present  influence  that  COMOPTEVFOR  has  on  the  TEMP,  and  on  "how 
much"  OT&E  he  wants  in  that  project.  Unfortunately,  it  is  sometimes  most 
subjective,  and  requires  pains  taking  negotiation  on  the  part  of  the 
Project  Manager.  If  there  is  too  much  OT&E  some  precious  DT&E  will  be 
given  up,  due  to  funding  constraints.  There  is  no  question  that  OT&E 
will  be  Integral,  early,  and  complete. 
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SECTION  V 


SELECTED  PROJECT  DATA 
Data  Collection 

This  author  was  principally  concerned  in  this  project  to  obtain 
current  data  from  Project  Managers,  to  iaentify  a "balance",  or  "im- 
balance", of  DT&E  and  OT&E  in  the  Naval  Air  Systems  Command  (NAVAIR). 

It  was  postulated  that  with  enough  sampling  of  carefully  selected  mission 
areas,  an  analysis  representative  of  the  overall  Navy  status  of  T&E 
would  result.  Project  Managers  who  served  previous  duty  with  OPTEVFOR 
test  squadrons,  was  also  a selection  criterion.  Fifty -seven  percent  of 
those  interviewed  had  this  experience,  the  remainer  had  been  Squadron 
Commanding  Officers,  or  had  extensive  Fleet  exposure.  Selected  questions 
were  prepared  for  common  base  data  collection  and  are  provided  in 
Appendix  A. 

Data  Base 

Balance  in  Testing? 

The  first  question  asked  of  the  interviewers  concerned  "balance"  of 
testing,  since  first  impressions  were  highly  desirable.  The  majority 
response  was  "there  is  a balance  of  OT&E  and  DT&E  testing",  but  two 
individuals  were  sure  that  there  was  no  such  balance,  and  that  DT&E 
was  being  driven  by  the  OT&E  side  of  the  house.  Test  planning  and  actual 
testing  were  performed  "to  satisfy  OPTEVFOR"  desires.  Additional, 
redundant  tests  were  performed  because  the  "test  data  flow  was  in  one 
direction  only",  ie.  OPTEVFOR  would  not  utilize  DT&E  test  data,  even 
though  they  were  a participating  member  of  the  flight  tests.  This  project 
manager  had  gone  through  three  Milestone  III  exercises  to  obtain  production 
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go-ahead  for  lot  procurement.  The  maturity  of  the  Project  Weapon  System 
is  an  important  factor  in  establishing  a balanced  T&E  process. 

Milestone  Acquisition? 

Considering  the  new  acquisition  milestone  procurement  process,  and 
its  impact  on  the  methods  of  T&E  in  the  Navy  that  existed  prior  to  1970, 
the  next  subject  centered  on  improvements  to  the  DT&E  process.  An  Initial 
Operational  Capability  (lOC)  date  had  been  slipped,  which  is  not  un- 
common, but  definitely  recommended  in  directives  governing  the  acquisition 
process.  The  Project  Manager  stopped  and  "fixed  it"  when  OPEVAL  discovered 
that  a changing  threat  would  require  adaptation  of  circuitry,  before 
OPEVAL  could  go  on  to  achieve  a production  go-adiead.  We  are  getting 
better  DT&E  evalmtions  but  with  the  same  amo’ont  of  testing,  even  less 
testing,  constrained  by  funding.  More  comprehensive  test  planning  as  the 
source  of  better  test  and  data  collection  for  the  dollar.  Another  major 
fact  is  that  "then"  we  used  to  do  "black  box"  testing,  and  now  it  is  full 
systems  testing.  A vast  change,  in  that  detailed  flight  test  planning  is 
mandatory  to  maximize  the  flight  test  period,  and  get  some  data  even  if 
partial  system  failure  occurs.  One  Project  Manager  is  facing  the  T&E  "GAP" 
(to  be  discussed  in  a later  section)  and  "that  impact  means  a great  deal 
of  sustaining  funding  to  a contractor,  to  wait  for  as  long  as  two  years 
after  production  go-ahead,  to  deliver  the  first  article".  It  means  a 
break  in  the  production  line,  and  that  could  cause  the  entire  project  to 
be  cancelled.  The  less-than  major,  project  manager  is  extremely  vulnerable 
to  the  DOD  5000.1  methodology  which  created  the  T&E  "gap".  He  can  be 
forced  into  the  "Gap"  when  his  funding  is  deferred  for  reprogramming  to 
the  major  project  in  trouble,  and  politically  sensitive.  Here  limited 
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production  go-ahead  is  mandatory  to  get  through  the  "Gap",  to  prevent 
excessive  sustaining  contractor  costs. 

OT&E  Impact? 

The  impact  on  DT&E  by  OT&E  emphasis  is  another  controversial  issue 
which  produced  varied  results.  No  one  likes  OPTEVPOR  looking  over  their 
shoulder  all  the  time,  but  it  is  a fact  of  life.  In  those  projects  that 
are  in  their  infancy,  the  COMOPTEVFOR  desire  to  have  DT&E  conceived  with 
the  OT&E  future  requirements  in  mind,  ie.  to  look  more  like  OT&E  can  be 
accomodated  and  areas  of  conflict  ironed  out  in  the  formative  stage. 
COMOPTEVFOR  has  even  agreed  to  share  data  and  strive  for  combined  testing 
by  mutual  agreement,  not  to  revisit  certain  areas  where  DT&E  will  be 
conducted,  thus  providing  the  data  for  OPEVAL.  This  is  a great  step 
in  the  right  direction  to  reduce  redundant  testing,  and  thereby  the  cost 
to  the  Project  Mamager.  Specifically,  the  number  of  test  articles  was 
reduced  60^  in  one  project,  when  highly  successful  DT4E  results  forced 
use  of  that  data  for  OT&E  evaluation.  On  the  other  hand,  COMOPTEVFOR 
tests  to  the  "now"  threat  when  the  system  was  designed  to  the  "then" 
threat.  Inevitably,  it  ends  up  in  the  COMOPTEVFOR  report  as  not  meeting 
"the"  threat. 

Sufficiency  of  DT&E? 

Adequate  DT&E  is  performed  for  NAVAIR  Weapons  Systems,  however  soft- 
ware testing  is  not  defined  properly;  "there  just  isn't  a good  ur>derstandlng 
of  what  is  involved  in  delivering  a "fully  capable"  operational  softwaie 
tape".  Software  OPEVAL  does  not  apply  to  "this"  weapon  system. 
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Cost  of  DT&E? 


DT&E  does  cost  too  much  in  the  minds  of  all  project  managers 
intervicKed,  but  to  put  this  idea  in  perspective,  "the  cost  is  about 
IQ^  too  much  not  50^ »"  "Our  overhaul  cost  are  too  high",  and  "we 
need  someone  to  help  us  get  more  for  our  testing  dollar."  The  cost  of 
DT&E  is  the  "nature  of  the  beast" , but  "we  were  able  to  cut  cost  by 
captive-carry  flight  testing"  for  missiles."  There  seems  to  be  a lack 
of  direction  so  the  project  maneiger  "does  the  best  he  can"  with  the 
dollars  he  is  able  to  keep  in  the  budget.  Some  solutions  include 
"front-end  loading  "the  T&E  process,  but  Congress  doesn’t  want  to  put 
that  much  money  into  the  front  to  save  as  much  as  six  or  seven  times 
the  initial  cost,"  in  the  later  stages  of  development.  Since  the 
early  design  stage  is  the  phase  most  critical  to  all  else  that  follows 
in  a project,  "we  should  do  a better  job  of  specification  tailoring  and 
put  more  time  into  planning  for  the  initial  contracting  effort,  before 
that  first  Request  for  Proposal  (RFP)  is  released  and  the  first  contract 
negotiations  commence.  A treatise  on  cost  of  testing  could  in  itself  be 
the  subject  of  a study  project,  but  for  the  sake  of  this  paper,  cost 
has  to  be  considered  as  more  than  an  equal  critical  driving  force  in  the 
acquisition  process. 

Combined  Testing? 

It  would  appear  that  combined  testing  to  some  Project  Managers  is 
realistic,  but  to  the  majority,  it  is  not.  The  definition  of  what 
constitutes  combined  testing  is  left  to  the  interpreter,  and  there  is  no 
"DOD"  definition  for  guidance,  however,  concurrent  testing  is  to  be 
avoided,  and  combined  testing  contemplated  only  in  special  case  situations. 


with  careful  segregation  to  retain  independent  OT&E.  Obviously,  a 
cost  savings  could  result,  but  NaVAIR  is  doing  very  little  combined 
testing  on  their  projects.  Combined  testing  is  not  realistic  when  one 
considers  the  philosophy  of  DT&E,  and  the  constraints  to  obtain  test  data 
by  careful  detail  test  planning  to  "get  the  most  out  of  the  test  dollar", 
ie.  the  desire  to  get  useful  and  necessary  "valid"  test  data  even  with 
partial  systems  failure.  OT&E  requirement  for  full  systems  capability 
is  the  dichotomy.  Further  discussion  can  be  found  in  the  other  sections. 

Emphasized  OT&E  - Better  DT&E? 

For  about  seven  years  now,  NAVAIR  has  operated  under  the  influence 
of  Increased  emphasis  on  OT&E.  The  attention  focused  by  all  DSARC 
reviews,  aind  political  pressures  from  all  sources,  cannot  but  cause 
improved  DT&E.  "You  can't  avoid  the  subject,"  although  one  may  be  con- 
cerned with  what  could  be  termed  more  critical  aspects  or  areas  for  con- 
centration in  the  Weapon  System  Acquisition  Process.  T&E  is  a hl^  cost 
driver  and  demands  critical  attention  at  the  earliest  point  in  project 
design  and  development.  New  ways  to  test  may  be  needed  for  the  evolving 
technology,  but  will  go  unnoticed  until  too  late  for  correction,  when  a 
Project  Manager  "trades-off"  adequate  T&E  planning.  "We  have  formalized 
our  methodology"  and  do  more  "contingency  planning",  as  a result  of  this 
influence . 

Approach  to  DT&E? 

NAVAIR  Project  Managers  agree  that  a change  must  take  place  in  "How 
we  presently  approach  Development  Test  and  Evaluation."  Hardware  testing 
has  been  and  will  continue  to  be  "rigid",  but  we  need  "new  concepts"  to 
approach  the  software  testing,  which  is  becoming  more  and  more  critical. 


as  imbedded  computers  manage  the  subsystems  employment  in  the  operational 
scenario.  Software  programs  have  to  be  "grown",  and  that  usually  takes 
more  time  and  test  effort  for  "debug"  to  finalize  each  module,  and 
deliver  an  "operational"  program.  Reliability  and  maintainability 
are  important  concepts  to  "inject"  at  the  earlist  possible  point  in 
design  of  any  weapon  system.  But  particularly,  reliability  is  the 
parameter  that  will  in  the  end,  "make  or  break"  the  project.  With  early 
stress  and  the  attendant  funding  necessary  to  design  for  more  reliability, 
an  "earlier  laboratory  demonstration"  can  be  scheduled,  which  gives 
NAVAIR  an  "earlier  look"  at  each  component.  Mean -Time -Between -Failure 
is  the  measure  of  reliability,  but  historically  what  "we  design  for", 
even  with  successful  laboratory  demonstration  of  design  criteria,  is 
never  what  "we  achieve  in  the  operational  world."  Reliability  "has  to 
be  designed  in  up  front,"  for  it  is  too  late  to  test  for  reliability  in 
the  stage  when  an  integrated  prototype  has  been  built. 

Milestone  "0"  and  Zero  Base  Budgeting? 

Milestone  Zero  was  instituted  by  DOD  Instruction  5000*1»  January  18, 
1977  and  it  has  been  stated  that  the  need  for  this  additional  milestone 
was  to  prevent  too  many  programs  from  progressing  to  Milestone  III,  and 
because  they  are  not  "truly"  ready  for  a full  production  go-ahead,  the 
resultant  was  a progression  of  additional  milestones,  ie.  Ill  A,B,C  etc., 
to  sustain  the  production  line  and  deliver  weapon  systems  to  the  user. 

The  Mission  Element  Need  Statement  (MENS)  that  is  required  to  be  prepared 
by  the  component  for  submission  at  Milestone"0" , will  hopefully  help  to 
prevent  the  wrong  or  mission  "enhancing"  project  from  the  start.  Coupled 
with  the  Presidents  direction  for  Zero  Base  Budgeting  (ZBB),  which  requires 
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identification  and  a prioritization  of  necessary  future  weapon  systems 
matched  to  funding  constraints,  the  two  concepts  sho\ild  prevent  buying 
systems  that  we  don't  need  or  that  do  not  meet  the  threat.  But  what 
effect  can  this  have  on  DT&E?  In  the  early  phase,  ie.  Basic  Research 
and  Elxploratory  Development,  "ZBB  is  a way  of  life,"  "we  have  been  doing 
it  all  along,"  in  fact,  "you  might  say  DT&E  is  ZBB."  Comprehensive  T&E 
planning  will  negate  auiy  effect  on  DT&E.  "ZBB  will  force  exposure  of 
those  programs  that  get  past  milestone  "0"."  Milestone  "0"  will  cause 
"better  requirements  definition  where  they  do  not  exist."  Many  Navy 
projects  just  don't  have  an  Operational  Requirement  (OR)  document,  or 
a very  inadequate  one  in  some  cases.  Detailed  requirements  are  left  to 
be  presented  in  subsequent  documents,  which  causes  subjective  inter- 
pretation and  departure  down  the  wrong  road.  Hopefully  the  MENS  will 
require  that  operational  requirements  are  clearly  defined  and  will  there- 
fore permit  better  criteria  for  DT&E  requirements. 

Control  of  DT&E? 

The  Project  Manager  has  to  have  control  of  DT&E.  Some  schools  of 
thought  consider  the  creation  of  a new  T&E  Czar  who  would  control  all 
Service  T&E,  and  the  budget  to  go  with  it.  In  fact,  there  is  a Govern- 
ment Accounting  Office  (GAO)  study  which  entertains  such  an  idea,  and 
even  considers  breaking  it  away  from  the  Services.  It  is  this  authors 
opinion  that  such  action  should  not  take  place.  The  creation  of  euiother 
"layer"  of  control  cannot  be  cost  effective  and  would  most  likely  not 
survive  present  Executive  Administration  scrutiny.  To  be  specific  in 
the  definition  of  control  of  DT&E  consider  the  "golden  rule"  that  he  who 
controls  the  "gold"  makes  the  "rules",  and  that  would  be  the  Project 
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Manager,  but  truly  in  the  end  analyses,  it  is  the  Congress. 

COMOPTEVFOR  Independence  and  Contributions? 

Since  OPTEVFOR  was  established  it  has  been  an  independent  test 
organization.  Their  charter  and  position  established  by  CNO,  support 
this  issue.  Some  Project  Managers  wish  they  were  not  "so"  Independent 
since  there  is  a high  cost  directly  attributed  to  this  factor,  and  it 
impacts  the  whole  DT&E  process.  Once  the  OT&E  approach  is  established 
by  COMOPTEVFOR  and  testing  philosophy  is  digested,  DT&E  has  to  be  planned 
and  performed  to  "Counter  balance"  OT&E.  The  OT&E  or  OPEVAL  reports  are 
a significant  lever  to  hold  over  the  Project  Manager  as  he  conducts  his 
DT&E  plan,  but  it  is  this  OPEVAL  report  that  causes  most  "heartache"  for 
the  Project  Manager.  Those  in  the  Navy,  especially  Project  Managers  who 
have  been  in  OPTEVFOR  jobs,  or  are  familiar  with  OPTEVFOR  reporting,  find 
little  trouble  in  the  interpretation  of  the  data  presented.  They  usually 
know  how  to  counter,  if  necessary,  with  DT&E  hard  technical  data  support, 
but  find  little  resource  countering  on  "outside  the  Navy"  source  who 
use  OT&E  reporting  for  their  own  political  position,  or  just  do  not  have 
a full  understanding  of  what  the  intended  purpose  of  the  reporting 
represents.  In  the  case  of  the  former,  OPTEVFOR  contributes  to  the  DT&E 
process  in  a positive  manner,  but  for  the  latter  it  is  all  negative.  The 
Project  Manaiger  must  become,  or  be  made  aware  of  the  impact  any  COMOPTEVFOR 
report  will  have  on  his  project. 

TEMP  Usefulness? 

The  variance  of  attitudes  on  the  subject  of  a TEMP,  and  the  measure 
of  its  use  or  application  is  of  considerable  Interest.  There  are  few 
projects  that  were  conceived  under  the  new  Milestone  Procurement  concept 
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that  have  been  completed;  many  others  were  started  prior  to  the 
Implementation  date  of  DOD  instruction  5000.1,  which  in  their  later 
stages  were  administered  using  its  precepts;  and  fewer  less  that  have 
been  subjected  to  the  Navy  emphasis  on  TEMP  requirements.  "Why  do  I 
need  a TEMP  for  this  project  when  the  production  line  will  be  closed 
next  March?"  "When  the  only  testing  to  be  done  is  Follow-On  Test  and 
Evaluation  (FOT&e).  COMOPTEVFOR  should  prepare  the  TEMP."  "Who  needs 
one  more  historical  planning  document"  when  we  describe  T&E  in  all  budget 
documents,  and  in  the  Decision  Coordination  Paper  (DCP),  which  is  the 
key  briefing  document  for  Defense  System  Acquisition  Review  Council 
(DSARC)  decision  making  at  each  major  milestone.  "I  used  the  TEMP  as 
an  integral  part  of  the  Request  for  Quote  (RFQ)."  It  defined  all  the 
necessary  testing  ”we  could  identify  at  this  time,  " and  shows  the  re- 
sponding contractors  the  scope  of  testing  that  the  government  will  require. 
"I  am  sure  we  would  not  have  considered  that  a second  set  of  support 
equipment  would  be  required."  "The  additional  cost,  and  potential  time 
lost  in  the  testing  process,  will  more  than  pay  for  the  TEMP  effort." 

"I  can  use  the  TEMP  to  set  Project  office  policy  on  T&E  2uid  resolve  areas 
of  conflict  with  COMOPTEVFOR  early  in  the  process."  The  TEMP  becomes  a 
contract  between  the  Developing  Agency  (DA)  and  COMOPTEVFOR,  when  the 
various  differences  are  negotiated  in  the  formative  stages,  then  it  requires 
COMOPTEVFOR  adherence  once  CNO  approval  is  obtained.  Project  design 
problems  cannot  be  designed  out  by  the  TEMP,  but  formal  agreements  of 
test  plans  can  be  achieved. 

OP-9e3/OPTEVFOR  Comments 

Time  did  not  permit  a trip  to  OPTEVFOR,  therefore,  OP-983  personnel 
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were  interviewed  to  obtain  data  relevant  to  this  project.  A question 
bank  was  prepared  as  provided  in  Appendix  A. 
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The  interplay  between  Project  Managers  and  OPTEVPOR  personnel  is 
"personality  oriented,"  ie.  the  "strong  beget  the  strong,"  and  the  lime- 
light projects  not  only  get  the  attention,  but  are  protected  idien  critical 
issues  center  on  their  existence  and  siurvival.  "We  don’t  hear  about  or 
get  very  much  Involved  in  those  projects  with  few  problems."  The  Budget 
process.  Five  Year  Defense  Plan  (FYDP),  and  Program  Operating  Memorandum 
(pom)  are  "out  of  phase"  with  Milestone  Procurement.  Project  Managers 
just  cannot  meet  the  "windows,"  but  DOD  allows  about  "10%"  when  the 
DSARC  process  requires  DCP  staffing  for  test  results.  The  "line  must  be 
kept  open"  as  the  alternative  price  to  be  paid  for  sustaining  a contractor 
to  sit  idle  through  the  "Gap"  is  not  acceptable,  especially  when  the  risks 
have  been  satisfied  to  an  acceptable  measure.  The  question  stands  out! 
Just  what  does  it  take  to  obtain  a production  go-ahead  at  Milestone  III? 
The  answer  can  be  found  only  in  a unique  set  of  rationale  for  each  project 
when  the  decision  day  arrives.  Historical  information  and  lessons  learned 
only  serves  the  avoidance  side  of  a Project  Managers  approach;  the  balance 
must  be  comprised  of  his  ability,  talents,  and  if  one  might  accept,  his 
charisma  for  the  "given"  decision  scenario. 

A balance  of  DTAE  and  OT&E  is  achieved  by  "each  side  pulling  in  their 
own  direction,"  and  sometimes  OT&E  "comes  up  short"  because  OPEVAL  occurs 
at  the  end  of  a program,  which  is  due  to  OT&E  phase  III  being  cut  short, 
while  awaiting  fixes  from  DT&E  phase  III.  Balance  can  also  be  achieved 
through  the  process  of  preparing,  changing,  and  obtaining  approval  of  the 
TEMP.  At  this  time,  considering  that  OPNAV  Instruction  3960.10  has 
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directed  implementation  to  have  been  accomplished  for  all  projects  cited 
for  compliance,  only  60  TEMP's  are  " in -hand" , as  recognized  by  OP-983. 
The  TEMP  is  a "historical"  not  a working  document. 

Combined  testing  can  be  cost  effective,  but  concurrency  should  be 
avoided.  Once  again  there  seems  to  be  a semantics  problem  amd  lack  of 
definition,  when  we  discuss  "data  sharing",  and  how  the  data  should  flow 
DT&E  to  OT&E,  but  this  sharing  of  data,  at  this  juncture,  is  a one-way 
flow.  One  project,  the  F-18,  does  have  a combined  test  team,  and  the 
planning  to  accomodate  later  decisions.  The  approach  is  encouraging, 
but  only  am  effective  cost  tracking  system  will  prove  the  merits  of 
their  results.  Overtesting,  like  combined  testing  is  an  over-used  term, 
but  as  viewed  from  this  source,  sometimes  "we  do  too  much,  amd  at  other 
times  too  little."  The  analyses  is  just  too  subjective  to  be  of  value, 
however  when  considering  a fixed  system  introduction  date,  coupled  with 
the  removal  of  critical  funds,  the  result  is  obvious. 


SECTION  VI 


CONCLUSIONS 

1.  DT&E  and  OT&E  within  the  Naval  Air  Systems  Command  is  a 
"balanced"  process  with  an  orgeinizational  structure  that  satisfies  DOD 
direction.  Each  Navy  agency  has  its  directed  and  useful  function. 

Various  studies  have  been  conducted,  to  reduce,  combine,  or  eliminate 
these  agencies  in  au:i  effort  to  eliminate  "overtesting".  No  better  way 
of  doing  business  has  resulted  when  the  trade-off  and  impact  is  truly 
understood.  Dr.  Eugene  G.  Fubini  of  the  Task  Force  on  Test  and  Evalu- 
ation, Defense  Science  Board,  reported  to  DDR&E  1?  February  1977  (also 
the  Fubini  Report)  thatt 

The  System  serves  the  evaluation  function  adequately 
by  providing  a solid  basis  for  a decision  to  proceed 
when  the  test  results  are  generally  satisfactory. 

Iliere  appears  to  be  little  or  no  overtesting  done 
under  the  directives;  what  testing  is  done  con- 
tributes its  full  value  to  the  improvement  and 
verification  of  system  performance. 

The  process  of  developing  a TEMP  for  NAVAIR  projects,  provides  the  basic 
"tool"  to  ensure  a balanced  TisE  effort,  and  if  commenced  at  the  earliest 
appropriate  time,  will  reduce  the  impact  and  cost  growth. 

2.  NAVAIR  Project  Managers  adhere  to  the  Milestone  Procurement 
concept,  which  in  reality  has  not  changed  the  Navy  T&E  methodology  that 
was  in  effect  prior  to  1970.  Improved  DT&E  has  resulted  for  the  testing 
"dollars"  that  are  available.  If  he  is  an  outstanding  manager  and  planner 
he  will  obtain  the  significant  DT&E  test  results  to  match  the  OT&E  report, 
and  give  the  DSARC  the  necessary  decision  making  criteria  for  a Production 
Decision,  and  thereby  appro’al  for  Service  Use.  If  he  is  shy  in  his  test 
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results,  depending  on  the  degree  of  remaining  risk  and  OPEVAL  report,  he 
should  come  away  with  at  least  a limited  production  decision  to  meet 
long  lead  previous  Navy  budget  (FTDP  and  POM)  submissions.  If  he  can 
achieve  neither  of  these,  he  is  the  wrong  man  for  the  job  and  will  surely 
be  replaced  as  soon  as  possible. 

3.  The  "Fubini  Report"  identified  the  T&E  "gap"  that  can  develop 
between  the  end  of  DT4E/0T&E  (or  TECHEVAL/OPEVAL),  and  the  beginning  of 
FOT&E,  and  which  lasts  about  two  years.  "The  time  lost  in  the  maturing  of 
the  production  system,  and  the  cost  to  the  contractor  and  the  government 
from  stopping  of  hardware  construction  activities,  as  the  program  moves 
from  R&D  to  production,  are  highly  undesireable."  (l6il4)  "The  so  called 
Test  and  Evaluation  continues  to  exist."  (15»7)  The  alternatives  to 
avoid  this  "m^p"  are* 

1.  Plan  at  the  start  of  engineering  development  for 
additional  phase  of  testing  to  cover  the  T4E  gap. 

2.  Early  in  the  DT4E  effort,  defend  long  lead  time 
production  funding  and  seek  production  funds  for 
low  rate  pilot  production. 

3.  Simply  allow  the  gap  to  exist,  which  may  be 
preferred  when  the  effort  to  reduce  the  gap  would 
require  commitment  to  a very  large  percentage... 
of  the  expected  program  cost  before  T&E  assurance 
of  a successful  product  could  be  obtained.  (15»10) 

4.  The  cost  of  Test  and  Evaluation  is  excessive  and  there  is  room 
to  cut  costs  by  whatever  means  that  can  be  identified,  but  not  to  detract 
from  the  main  purpose  of  DT&E  and  OT&E.  Specification  tailoring,  and 
initial  contract  planning  are  significant  methods  to  develop  realistic 
design  to  cost  concepts,  but  should  be  implemented  before  the  first 
request  for  proposal  leaves  the  project  office.  A satisfactory  cost 
accounting  system  to  permit  an  audit  trail  for  test  costs,  does  not  exist. 
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Cost  Schedule  Control  System  Criteria  (CSCSC)  is  imposed  on  all  con- 
tractors for  weapon  system  in  the  major  category,  and  those  specially 
selected  by  DOD.  Some  fall  out  of  CSCSC  is  filtering  down  to  some 
government  agencies,  but  not  throughout  DOD.  Navy  Industrial  F\inding 
(NIF)  is  a step  toward  defining  total  cost  at  test  activities,  but  no 
"audit  trail"  and  only  limited  documentation  is  available. 

5.  Combined  testing  when  well  planned,  structured,  and  coordi- 
nated, with  common  data  sharing  based  on  responsibility  for  data 
r.-ollection  assignment,  will  drive  the  cost  of  testing  down.  The  con- 
cept of  TECHEVAL  followed  by  OPEVAL  must  be  retained  in  tact  to  retain 
the  independent  test  activity  concept.  The  TEMP  provides  the  key  to 
prevent  concurrency  when  a combined  testing  approach  is  selected.  Com- 
bined testing  within  the  project  is  not  deemed  realistic  by  Project 
Managers,  however,  when  different  projects  can  combine  their  test 
schedules  to  utilize  shared  resources  a distinct  cost  saving  will  result, 
eg.,  in  the  Anti-Submarine  warfare  test  scenario,  the  same  target 

submarine  could  be  used. 

6.  A change  has  to  take  place  in  the  NAVAIR  approach  to  DT&E,  in 
fact,  certain  changes  are  in  progress.  NAVAIR  started  a campaign  two 
years  ago  to  stress  reliability  and  maintainability  for  designated  projects. 
Adequate  funding  is  usually  lacking  when  it  is  most  needed,  ie.  in  the 
front  end  of  the  design  effort.  Congress  does  not  seem  to  realize  the 
merit  of  "front-end"  funding  hi^ly  technical  weapon  systems  projects, 

and  is  only  concerned  with  "this"  year  dollars.  With  the  paucity  of 
funds,  NAVAIR  performs  Reliability  Laboratory  Demonstrations  early  in 
the  development  phase.  Early  reliability  testing  for  validation  pro- 


vides  the  operational  avallabili+y  and  reduces  Operating  and  Support 

cost,  and  in  turn  life  cycle  costs.  DDR&E  emphasized  these  issues  in 

his  statement  to  Congress  that: 

. . .early  T&E  provides  an  indication  of  technical 
risk  and  potential  operational  value  of  a system 
prior  to  commitment  of  large  amount  of  R&D  funds. 

...provides  the  basis  for  choosing  from  among 
competing  lechnical  approaches  those  vrtiich  offer 
the  greatest  promise  if  further  developed.  (l3«X-4) 

Reliability  testing  is  another  area  of  principal 
current  emi^asis. 

We  are  requiring  Program  Managers  to  specify, 
prior  to  the  start  of  engineering  development, 
interim  reliability  thresholds  vrtiich  must  be 
attained  before  further  advancement  of  a system 
through  the  acquisition  cycle.  (13«X~5) 

. . .purpose  is  to  develop  procedures  which  will 
provide  necessary  incentive  at  the  very  start 
of  a program  for  attainment  of  satisfactory 
field  reliability.  A prime  example  is  the 
F-18  filter  program.  (l3«X-6) 

7.  Software  testing  is  causing  a stretch-out  in  the  DT&E  process 
which  requires  new  concepts  and  understanding.  Hardware  test  are  rigid 
but  software  has  to  grow  and  testing  has  to  change. 

8.  The  Milestone  Zero  requirement  for  a MENS  will  force  better 
Operational  Requirements  (OR)  definition. 

9.  The  control  of  DT&E  has  to  remain  in  the  hands  of  the  Project 
Manager,  and  Developing  Agency.  The  Project  Manager  does  not  tell  the 
test  agencies  how  to  test,  therefore,  the  test  agencies  control  the  actual 
conduct  of  DT&E  and  provide  check  and  balance  on  design  specifications. 

10.  OPTEVFOR  has  been  an  independent  test  agency  since  it  was 
established  and  will  continue  as  chartered.  OPTEVPOR  reports  have  not 
been  timely,  are  quite  often  too  qimlitative  or  subjective  in  content 


35 


and  sometimes  cause  unnecessary  "fire-drills."  The  "Fubini  Report" 

provides  the  following  information { 

...major  systems  decisions  are  judgements  based  on 
a wide  range  of  qxjalltative  considerations,  rather 
than  on  statistical  compilations,  and  the  limitations 
and  outcomes  of  the  operational  tests  must  be  compre- 
hensive and  meaningful  to  the  decision  makers  as  well 
as  the  testing  community.  (16j75) 

11.  The  TEMP  is  heavily  emphasized  in  the  Navy.  It  is  a useful 
control  document  that  DDR&E  considers  an  "important  tool"  in  conducting 
DSARC  reviews,  "a  significant  benefit... is  early  establishment  of  a 
close  working  relationship  between  the  development  and  T&E  agencies." 
(l3«X-4).  Most  NAVAIR  projects  respond  to  the  TEMP  requirement  due  to 
establishment  of  and  emphasis  stressed  by  AIR-06.  The  TEMP  becomes  a 
contract  between  DT&E  and  OT&E  parties  once  the  test  requirements  are 
negotiated.  CNO  approval  of  a TEMP  is  as  directed  by  appropriate 
instructions.  Preparation  of  a TEMP  for  new  projects  can  help  to 
identify  resources  that  may  be  overlooked  without  early  preparation. 

12.  OPTEVFOR  requires  production  systems  for  OPEVAL.  Project  Managers 
need  OPEVAL  to  get  a go-ediead  for  production.  Limited  production  proto- 
types are  therefore  the  only  test  articles  available. 

13.  The  Budget  cycle  and  Milestone  Procurement  process  are  in- 
consistant,  which  prevents  the  accomplishment  of  adequate  T&E  of  potential 
weapon  systems. 
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SECTION  VII 


RECOMMENDATIONS 

1.  An  important  aspect  of  the  potential  to  recognize  a reduction 
in  the  amount  of  T&E,  and  thereby  reduce  overtesting,  duplicative,  or 
redundant  T&E  to  a minimum,  is  valid  data  sharing.  DT&E  and  OT&E  data, 
including  validated  contractor  T&E  documented  data,  must  be  accepted 
zind  shared  throughout  the  T&E  process.  Early  identification  of 
necessary  data  i^r  future  evalu£i.tion  should  be  accomplished  in  the 
early  design  phase,  and  responsibility  assigned  for  collection  as 
mutually  agreed  to  by  DT&E,  OT&E,  BIS,  and  the  contractor  hierarchy. 

2.  The  DSARC  should  continue  to  support  the  need  for  long  lead 
funding  to  reduce  the  T&E  gap  whe:.  technical  risk  Is  reasonable,  and 
operational  requirements  can  be  met. 

3.  A "Design  to  Test"  concept  should  be  generated  within  DOD  and 
particularly  within  NAVAIR,  for  imposition  of  CSCSC  on  NAVAIR  field 
activities,  and  also  the  laboratories  who  conduct  NAVAIR  sponsored 
work.  CSCSC  can  be  integrated  with  NIF,  however  those  activities  such 
as  the  laboratories  should  also  be  brought  under  NIF  requirements, 
rather  than  present  methods  of  fund  accounting. 

4.  Project  Managers  should  devise  effective  methods  to  obtain  as 
much  funding,  as  is  humanly  possible,  early  in  the  project  design  phase, 
where  70S?  of  the  project  cost  is  frozen  in  the  conceptual  phase  of 
development. 

5.  The  NAVAIR  Assistant  Commander  for  Test  and  Evaluation  (AIR -06) 
charter  and  functions  should  be  emphasized  to  ensure  proper  test-tasking 
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assignments  to  the  field  activities  who  can  provide  more  cost  effective 
management  without  degradation  of  DT&E. 

6.  Test  resources  should  be  shared  by  DT&E  and  OT&E  communities 
whenever  the  opportunity  permits.  Back-up  flights  should  be  scheduled. 

7.  Combined  Testing  should  be  encouraged  and  persued  by  Project 
Managers  in  all  new  projects,  with  test  requirements  and  resources 
adequately  defined  in  the  TEMP. 

8.  COMOPTEVFOR  should  establish  a policy  of  conducting  more 
evaluation  effort  rather  than  test  effort,  based  on  data  sharing,  and 
acceptance  of  documented  valid  DT&E  test  data  that  is  applicable  to 
OT&E.  A role  of  participate  and  monitor  sliould  be  employed  until  the 
project  weapon  system  is  ready  for  OPEVAL.  Crew  sharing  arrangements 
should  be  established  with  responsible  assignments  and  test  vehicles 
could  be  transferred  from  DT&E  to  OT&E  activities  when  the  system  is 
representative  of  OPEVAL  Configuration. 

9.  Somehow  Project  Managers  have  to  "lobby"  for  more  funding  for 
design,  in  the  conceptual  stage,  and  validate  the  requirement  to  build 
the  reliability  up  front  where  it  is  needed. 

10.  Software  testing  and  its  effect  on  a go/no  go-ahead  for  pro- 
duction decision  Bhould  be  split  into  two  categories.  One  that  produces 
acceptable  software  programs  that  match  the  operation  of  the  hardware 
that  is  proven  and  acceptable  for  go-ahead,  and  the  second  category 
which  updates  the  "operational  employment"  of  subsystems,  which  may  be 
enhancements  and  need  additional  time  to  mature.  Two  ASU  decisions 
could  be  conceived,  one  coupled  to  the  hardware  and  the  second  to  an 
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improved  software  effort. 


11.  No  new  or  additional  top-level  T&E  agency  should  be  established. 

12.  COMOPTEVFOR  should  provide  more  timely  reports  for  decision 
making.  A quick  look  message  report  is  almost  mandatory  when  one 
considers  the  disparity  of  the  budget  cycle  and  Milestone  Procurement 
process. 

13.  Better  communication  has  to  be  developed  between  the  Project 
Manager  and  OPTEVFOR  to  prevent  surprises. 

14.  The  OPTEVFOR  report  should  describe  how  the  weapon  system 
countered  the  "then"  threat,  in  perspective  to  how  it  can  perform 
against  the  "now"  threat. 

15.  DDT&E  should  continue  to  emphasize  the  requirement  for  a TEMP 
for  DSARG  I and  subsequent  milestones. 

16.  Project  Managers  should  draft  a TEMP  and  make  it  a part  of 
the  RFQ,  RFP,  and  Contract.  If  not  approved,  the  intent  of  a TEMP 
should  be  included  in  these  documents. 

17.  COMOPTEVFOR  should  not  accept  any  system  for  OPEVAL  that  is 
not  representative  of  the  production  article. 

18.  COMOPTEVFOR  should  man  the  squadrons  with  pure  operational 
personnel,  rather  than  accept  Aeronautical  Engineering  Duty  Office  (aEDO) 
personnel  in  the  structure.  Their  tendency  is  to  revert  to  the  technical 
training  that  they  have  received,  and  develop  test  techniques  beyond  ^.le 
needs  of  OPEVAL.  Former  Project  Managers  should  be  assigned  billets  at 
the  Staff  0-6  level. 
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19.  Project  Meuiagers  and  other  DOD  officials  must  convince  the 
Congress  that  more  funding  is  necessary  early  in  the  design  phase  of 
each  project,  and  that  necessary  funding  for  T&E  is  available,  when 
it  is  required,  to  transition  through  critical  phases  of  development. 


APPENDIX  A 


Interview  Questions  for  Project  Manaigert 

1.  Is  there  a balance  of  DT&E  and  OT&E  In  NAVAIR  projects? 

2.  Are  we  doing  business  any  differently  than  before  Milestone 
Acquisition  was  imposed? 

3.  Is  there  an  impact  on  DT&E  by  OT&E  emphasis? 

4.  Do  we  do  adequate  DT&E? 

5.  Are  we  paying  too  much  for  DT&E? 

6.  Can  we  reduce  the  cost  of  DT&E?  By  what  means? 

7.  Is  combined  DT&E  and  OT&E  a realistic  approach? 

8.  Do  you  realize  any  additional  cost  in  combined  testing? 

9.  Has  emphasized  OTE  improved  the  DT&E  methodology? 

10.  Should  our  present  approach  to  DT&E  be  changed? 

11.  What  effect  will  Milestone  "0"  have  on  DT&E? 

12.  Will  "0"  base  budgeting  affect  DT&E? 

13.  Who  should  have  control  of  DT&E?  Who  does? 

14.  Do  you  feel  OPTEVFOR  is  an  independent  Test  Agency? 

15.  When  OPTEVFOR  performs  OTE  do  they  contribute  plus  or  minus 
factors? 

16.  Is  the  TEMP  a useful  management  tool  or  is  it  just  required? 
What  effect  does  it  have  on  cost,  schedule,  and  performajice? 

Interview  Questions  For  OPTEVFOR/OP-983  Personnel! 

1.  Is  there  a balance  of  DT&E  and  OT&E  in  NAVAIR  Weapon 
Systems  Acquisition? 

2.  How  would  you  describe  the  present  PMA/OPTEVFOR  relationship? 

3.  Do  any  significant  interface  problems  exist? 
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4.  Is  the  current  Navy  Instruction  3960.10  ad.eqxa.te  to  meet  the 
requirements  for  T&E? 

5.  In  your  opinion,  is  combined  testing  a reasonable  approach? 

Are  we  over  testing  in  either  the  Developmental  or  Operational 
environment? 
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